Rejoinder

In view of my discussion of the paper by Wallace and Dowe with an outline of the minimum description length (MDL) principle, of which the minimum message length (MML) principle is seen to be a primitive, restricted and inferior implementation, I do not wish to go into a detailed rebuttal of the numerous issues raised in Professor Wallace’s lengthy discussion. Rather, I will summarize the salient features of the MDL principle which still appear to cause confusion and address only the relevant points also raised by the other discussors. The MDL principle requires two things: one or more suggested classes of models as probability measures or codes and the observed data. The objective is to calculate a yardstick in terms of the length of a code for the data, provided by the model classes, by which they and the models in them can be compared and to find the best in light of the data. Hence, the task of selecting the collection of the classes themselves lies beyond the principle. Accordingly, as should be clear, the shortest code length found will depend, in addition to the data, also on the model classes suggested. In order to avoid the non-computability problem of finding the shortest code length, relative to each class, none of them must include the set of all computable models. This, however, creates the difficulty that the idea of the shortest code length, called for by the principle, cannot be taken in the literal sense. Rather, it will be shortest in certain probabilistic senses, which amounts to being shortest for all typical strings generated by almost all models in each class considered. This suffices in practice. Prior knowledge is used in the principle to guide the selection of the suggested model classes, which may include a distribution for parameters, called a ‘prior’ in Bayesian analysis. However, unlike for Baysians, for whom the quality of such prior knowledge cannot be contested (for them there is no such thing as ‘bad’ or ‘good’ prior knowledge), its role in the MDL principle is only tentative: a prior is good if it helps to reduce the code length; otherwise it is worthless and should not be used. This is as it must be in the MDL principle, for admitting unqualified prior knowledge would create conflict with the principle. With this proviso the MDL principle does admit the use of prior information in the form of distributions for parameters. This can be accomplished either by replacing the ‘canonical’ prior in the normalized maximum likelihood (NML) code by any desired prior, or to calculate a mixture density, called often but misleadingly a ‘Bayesian mixture’, as discussed in Professor Clarke’s section 2, The Bayesian Connection. Its role in the MDL principle is to provide a short code length, which is not at all why it is used in Bayesian analysis. For instance, the justification of the mixture defined by Jeffreys’ prior is purely mathematical in the MDL principle in that it achieves the channel capacity, which is a notion linked with a short code length. While the NML model has its limitations, such as not defining a random process, it is perfectly well capable of handling Professor Wallace’s binomial protocol on Bernoulli trials. If the data consist of 469 trials with 100 successes, the NML density function defines an unambiguous ideal code length for the string. The fact that the length 469 is not fixed requires a small adjustment. The simplest way to do this is to add log 469 to the result. Again, if 3000 trials are included as an upper bound for the length, it obviously does not change the data and that ‘prior’ knowledge should be ignored just as any other irrelevant knowledge such as the belated report of the person who did the trials. In general, the various restrictions required in the NML model define different model classes and the best among them can be found by the MDL principle. In fact, repeating the same normalization process to the various hyperparameters, including the number of the primary parameters, leads to a new and powerful criterion for the basic linear-Gaussian regression problems, including the so-called denoising problem, which cannot be found nor matched by Bayesian nor orthodox statistical techniques; see [1]. This also provides a partial answer to Professor Dawid’s worry about the practical pay-off of the MDL principle. There are many other successful applications, but let me turn his argument about reaching essentially identical conclusions by more standard methods around and state that, indeed, in order to match the results obtainable with this single clear-cut principle the entire stockpile of the standard methods, many of them ad hoc, developed over decades by the collective wisdom and experience of eminent statisticians, does not appear to be enough. The MDL principle as stated involves a class of probability measures as models and the associated yardstick in terms of code length or, equivalently, probability assigned to the given data. All but trivial data are in effect random; i.e. not generated by a fixed algorithm and the very purpose of model building is to extract the regular and learnable features from the random ‘noise’ in the data, which is how the MDL principle breaks up the data. Moreover, there is no need to make the impossible-to-justify assumption that the data be samples from some population and the